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—=*  - This  technical  report  describes  a decision-aiding  technique  developed 
to  assist  division- level  commanders  and  their  staffs  in  choosing  among 
alternative  courses  of  action  for  tactical  engagement.  By  employing 
principles  of  multi-attribute  utility  assessment  methodology,  a two-level 
model  consisting  of  five  general  categories  (terrain,  own  forces,  enemy 
forces,  weather,  and  risk)  and  twenty-four  factors  was  developed  and 
implemented  on  an  IBM  5100  computer.  In  applying  the  decision  aid,  the 
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juser  Is  required  Co  score  each  of  Che  alcernaclve  courses  of  acclon  on 
each  faccor  and  Co  assign  a welghc  indlcaclng  Che  importance  of  each  faccor 
In  dlscrlainaClng  among  Che  alcernaclves.  A simple  algorlchm  is  used  Co 
calculaCe  a weighted  score  for  each  course  of  action,  Che  highest  score 
being  an  Indication  of  Che  preferred  course  of  acclon.  A sensitivity 
analysis  provides  a measure  of  the  robustness  of  the  scores  and  weights 
assigned  by  the  user. 

The  aid  was  designed  to  provide: 

1.  A checklist  of  the  salient  factors  that  need  to  be  considered 
in  choosing  among  alternative  courses  of  action,  thus  at  least 
inviting  detailed  consideration  of  the  complete  spectrum  of 
factors  that  bear  on  tactical  choice. 


A logical  method  for  integrating  subjective  elemental  assess- 
ments into  an  indication  of  cholcey  To  the  extent  that  the 
elemental  (factor)  judgments  are-^alid,  the  preferred  course 
of  action  thus  derived  ^houldbe  the  optimal  choice  given  the 
way  the  decision-aakeiT  has  valued  courses  of  action  on  the 
varioue-'factors . 


TCmeans  of  well-focused  communication  about  the  decision 
problem  at  hand,  by  making  the  decision  problem  and  value 
structure  explicit  and  by  requiring  judgments  in  quantitative 
rather  than  qualitative  terms. 
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SUMMARY 


This  technical  report  describes  a decision-aiding  tech- 
nique developed  to  assist  division-level  commanders  and 
their  staffs  in  choosing  among  alternative  courses  of  action 
for  tactical  engagement.  By  employing  principles  of  multi- 
attribute utility  assessment  methodology,  a two-level  model 
consisting  of  five  general  categories  (terrain,  own  forces, 
enemy  forces,  weather,  and  risk)  and  twenty-four  factors  was 
developed  and  implemented  on  an  IBM  5100  computer.  In 
applying  the  decision  aid,  the  user  is  required  to  score 
each  of  the  alternative  courses  of  action  on  each  factor  and 
to  assign  a weight  indicating  the  importance  of  each  factor 
in  discriminating  among  the  alternatives.  A simple  algorithm 
is  used  to  calculate  a weighted  score  for  each  course  of 
action,  the  highest  score  being  an  indication  of  the  preferred 
course  of  action.  A sensitivity  analysis  provides  a measure 
of  the  robustness  of  the  scores  and  weights  assigned  by  the 
user. 


The  aid  was  designed  to  provide: 

1.  A checklist  of  the  salient  factors  that  need  to  be 
considered  in  choosing  among  alternative  courses 
of  action,  thus  at  least  inviting  detailed  con- 
sideration of  the  complete  spectrum  of  factors 
that  bear  on  tactical  choice. 

2.  A logical  method  for  integrating  subjective  ele- 
mental assessments  into  an  indication  of  choice. 

To  the  extent  that  the  elemental  (factor)  judg- 
ments are  valid,  the  preferred  course  of  action 
thus  derived  should  be  the  optimal  choice  given 
the  way  the  decision  maker  has  valued  courses  of 
action  on  the  various  factors. 


3. 


A means  of  well-focused  communication  about  the 
decision  problem  at  hand,  by  making  the  decision 
problem  and  value  structure  explicit  and  by  re- 
quiring judgments  in  quantitative  rather  than 
qualitative  terms. 

Since  all  training  is  basically  concerned  with  shaping 
the  trainee's  internal  value  structure,  the  property  of  the 
decision  aid  in  making  that  subjective  structure  visible  in 
detail  should  be  of  value  in  training  contexts  concerned 
with  tactical  decision  making. 


FOREWORD 


The  Human  Factors  Technical  Area  or  the  Army  Research  Institute  is 
concerned  with  the  human  resource  demands  of  increasingly 
complex  battlefield  systems  used  to  acquire,  transmit,  pro- 
cess, disseminate,  and  utilize  information.  This  increased 
complexity  places  great  demands  upon  the  operator  interacting 
with  the  machine  system.  Current  research  focuses  on  human 
performance  problems  related  to  interactions  within  command 
and  control  centers,  as  well  as  on  issues  of  system  develop- 
ment. It  is  concerned  with  such  areas  as  software  develop- 
ment, topographic  products  and  procedures,  tactical  symbology, 
user-oriented  systems,  decision  making,  systems  integration, 
and  utilization. 

An  area  of  special  interest  is  the  exploitation  of 
tactical  data  systems  to  assist  the  tactical  commander  and 
staff  in  analysis  and  decision  making.  One  approach,  used 
in  the  current  effort,  is  to  adapt  the  formal  methods  of 
decision  analysis  to  the  tactical  decision  process.  A 
decision  support  module  was  developed  to  assist  users  in 
evaluating  alternative  friendly  courses  of  action.  A pre- 
liminary evaluation  conducted  at  Fort  Leavenworth,  Kansas, 
was  extremely  favorable,  and  recommendations  were  made  for 
field  and  classroom  implementation. 

Research  in  the  area  of  decision  support  is  conducted 
as  an  in-house  effort  augmented  contractually  by  organizations 
selected  for  their  unique  capabilities  and  facilities  for 
research  in  this  area.  This  effort  is  responsive  to  the 
requirements  of  Army  Project  2Q762722A765,  as  well  as  to  the 
general  requirements  of  HRN  77-296  (ADP  Methods  for  Utiliza- 
tion of  Analytic  Aids  and  Logic  Mouels  in  Intelligence  Pro- 
cessing) and  HRN  78-151  (Processing  and  Problem  Solving  Aids 
in  Tactical  Systems) . 
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1.0  INTRODUCTION 


Technological  advances  in  weaponry  and  associated 
support  systems  have  greatly  increased  the  complex  of  factors 
which  must  be  taken  into  account  in  tactical  decision  making. 
While  decision  acumen  has  long  been  of  paramount  importance 
in  combat,  in  modern  warfare  the  speed  with  which  forces  can 
be  committed  to  action  and  the  enormous  lethality  of  arms 
have  created  a circumstance  where  disaster,  rather  than 
increased  losses  or  a temporary  setback,  is  a much  more 
likely  consequence  of  poor  tactical  decisions.  These  factors, 
along  with  compelling  research  evidence  (References  3,  5, 

12,  14)  pointing  to  profound  human  fallibilities  in  even 
simple  decision  tasks,  have  served  as  the  impetus  for  a 
search  for  ways  to  aid  commanders  and  their  staffs  in  the 
extremely  complex,  high-stake  decision  situations  they  face. 

Evaluating  and  choosing  among  alternative  courses  of 
action  in  tactical  situations  is  an  example  of  what  is  often 
an  extremely  complex,  difficult,  and  critical  decision  task, 
and  one  that  is  the  central  focus  of  the  decision-aid  develop- 
ment described  in  this  report.  As  presently  conducted,  the 
assessment  of  alternative  courses  of  action  requires  that 
the  commander  hold  in  mind  the  essential  details  of  friendly 
action  options,  those  of  the  enemy,  along  with  a host  of 
factors  bearing  on  the  effectiveness  of  each,  while  conducting 
a mental  war-game  of  the  action  and  results  that  would  ensue 
from  each  option.  The  course  of  action  deemed  most  appro- 
priate in  this  rather  global  thought  process  becomes  the 
selected  option. 

While  skilled  decision  makers  do  a remarkably  good  job 
in  these  judgmental  decision  roles,  considering  their  com- 
plexity, experimental  evidence  demonstrates  that  in  proba- 
bilistic, multi-attributed  decision  tasks  such  as  these. 
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many  fallibilities  of  judgment  often  come  into  play.  Rele- 
vant attributes  may  be  overlooked,  probabilistic  inferences 
may  be  faulty,  and  the  integration  of  information  over  many 
relevant  decision  dimensions  may  not  be  logically  consistent 
with  either  the  objective  situation  or  the  decision  maker's 
subjective  evaluation  of  the  objective  information. 

Decision  analysis,  an  applied  derivative  of  statistical 
decision  theory,  is  a methodology  developed  to  help  decision 
makers  avoid  these  judgmental  pitfalls  and  to  aid  them  in 
reaching  logically  consistent,  rational  choices.  The  method- 
ology is  fully  described  in  a number  of  sources  (References 
4,  7,  8,  11,  13).  In  brief,  it  represents  a divide-and- 
conquer  approach:  Complex  decision  problems  are  disaggregated 
into  the  relevant  component  or  elemental  factors;  judgments 
(valuations)  are  rendered  by  the  decision  maker  on  these 
elemental  factors  (which  presumably  are  cognitively  easier 
for  the  decision  maker) ; these  elemental  valuations  are  then 
aggregated  via  formal,  logical  algorithms  into  an  indication 
of  optimal  choice. 

In  the  decision  aid  development  described  in  this  paper, 
one  aspect  of  decision  analysis,  multi-attribute  utility 
assessment  (MAUA) , was  used  as  the  methodological  base. 

This  method  is  summarized  and  fully  treated  in  References  6 
and  7.  In  brief,  MAUA  is  a method  for  deriving  a single 
index  of  worth  for  an  object  of  assessment  having  multiple 
and  often  conflicting  dimensions  of  value.  Possible  courses 
of  action  for  an  impending  military  engagement  are  examples 
of  such  multi-attributed  alternatives.  One  course  of  action, 
for  example,  might  be  superior  in  terms  of  avenue  of  approach, 
and  the  degree  to  which  it  exploits  fields  of  fire,  but  may 
be  poor  in  terms  of,  say,  mobility  and  logistic  support  pro- 
visions. It  is  often  the  case,  as  in  the  example  just 
given,  that  the  many  relevant  attributes  have  no  objective 
metric;  and  it  is  usually  the  case  that  a common  metric 
applicable  to  each  of  the  relevant  attributes  is  lacking. 


In  the  MAUA  approach,  this  measurement  problem  is 
handled  by  the  concept  of  subjective  utility — an  expression 
of  subjective  value.  In  applying  MAUA,  the  objects  of 
assessment  (in  this  case,  alternative  courses  of  action  for 
tactical  engagement)  are  disaggregated  into  their  relevant 
major  attributes  (factors).  An  expression  of  the  decision 
maker's  utility  for  the  contributions  of  each  of  the  attri- 
butes to  overall  performance  is  elicited.  These  utilities 
are  then  aggregated  by  appropriate  algorithms  to  yield  an 
overall  index  of  worth  (utility)  for  each  of  the  options 
under  consideration.  Thus,  MAUA  is  a means  of  systematically 
and  logically  combining  subjective  assessments  into  overall 
indices  of  worth. 

It  should  be  pointed  out  that  all  decisions  are  matters 
of  subjective  judgment.  Even  with  perfect  information,  the 
critical  interplay  between  information  and  the  subjective, 
internal  value  structure  of  the  decision  maker  is  the  true 
locus  of  decision.  The  MAUA  approach  makes  those  subjective 
assessments  explicit  attribute-by-attribute  and  provides  a 
formal  logic  for  aggregating  the  elemental  judgments  into  an 
indication  for  choice.  The  use  of  subjective  judgments  in 
MAUA  is  in  no  way  a substitute  for  hard  objective  data. 

Rather,  the  approach  captures,  in  a systematic  fashion,  the 
judgmental  implications  of  whatever  relevant  information  is 
at  hand. 


2.0  THE  DECISION  PROBLEM 


The  decision  aid  development  effort  covered  in  this 
report  focused  specifically  on  the  problem  of  choosing  the 
best  course  of  action  from  a set  of  alternative  courses  of 
action  for  tactical  engagement.  The  focus  was  further  con- 
strained to  division-level  operations,  though  the  decision- 
aiding  model  that  was  developed  is  probably  appropriate  to 
other  command  echelons  as  well. 

As  taught  in  the  Army  Command  and  General  Staff  College, 
the  decision  process  for  evaluating  and  choosing  among 
alternative  courses  of  action  entails  the  following  steps 
(Reference  1)  : 

1.  Given  a corps  order  assigning  a mission,  identify 
specif ieu  and  implied  tasks. 

2.  Restate  the  mission  as  necessary  to  ensure  under- 
standing . 

3.  Gather  and  analyze  all  relevant  data. 

4.  Make  logical  assumptions  in  areas  where  data  is 
unavailable . 

5.  Conduct  mini-analyses  of  the  situation  and  mission 
and  formulate  alternative  courses  of  action. 

6.  Assess  the  alternative  courses  of  action  by  war- 
gaming each  through  the  impending  action  and 
choose  that  which  fares  best. 
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A number  of  general  points  should  be  made  with  regard 
to  the  six  steps  presented  above.  First,  it  is  recognized 
that  the  steps  are  an  approximation  to  a formal  school 
model.  In  field  applications,  one  is  certain  to  find  many 
variants  of  and  alternatives  to  the  six-step  decision  pro- 
cess. These  will  be  driven  largely  by  the  commander's 
"style”  and  the  time  available  to  make  a choice.  Nonethe- 

i 

less,  however  approached,  the  intricate  array  of  factors 
that  must  be  considered  in  reaching  a choice  make  a formi- 
dable decision  task  indeed.  Secondly,  while  all  the  steps 
outlined  above  are  important,  steps  five  and  six,  concerned 
with  the  formulation  of  courses  of  action  and  the  assessment 
of  them,  are  steps  that  entail  the  level  of  decision  com- 
plexity that  often  leads  to  less  than  optimal  choices.  We 
will  have  little  to  say  about  the  crucial  step  of  formulating 
courses  of  action  beyond  noting  in  passing  that  this  complex 
step  bears  very  heavily  on  the  quality  of  the  choice  that 
will  eventually  be  made.  If  a "best"  option  is  not  formu- 
lated as  a candidate  for  selection,  it  obviously  cannot  be 
selected  by  any  methodology.  Our  attention  focused  solely 
on  assessing  and  choosing  among  alternative  courses  of 
action  once  these  were  formulated.  In  this  endeavor,  we 
assumed  that  the  alternatives  posed  were  all  serious  contend- 
ers when  initially  formulated  rather  than  a lead  candidate 
with  weak  alternatives  thrown  in  for  exercise  purposes.  In 
the  latter  case,  decision  aiding  would  hardly  be  required. 

Turning  attention  now  to  the  sixth  step,  war-gaming  the 
alternative  courses  of  action  to  select  the  best  One,  con- 
sider the  information  burden  and  mental  integration  involved. 
The  decision  maker  must  hold  in  mind  all  potential  friendly 
courses  of  action,  as  well  as  those  open  to  the  enemy;  in 
addition,  there  are  a host  of  factors  bearing  on  the  exe- 
cution of  all  of  these  possible  actions.  With  all  this  held 
in  mind,  the  decision  maker  (commander)  plays  out  a mental 
war-game,  testing  each  course  of  action  to  reach  a decision 
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a 3 to  which  is  best.  Even  if  one  invokes  concepts  of 
"chunking"  and  other  methods  involving  substitution  of 
concepts  for  detail,  it  should  be  evident  that  the  required 
decision  can  be  an  extremely  complex  and  difficult  matter. 
Research  strongly  supports  the  view  that  if  people  must 
contend  with  more  than  five  to  seven  factors  simultaneously, 
sometimes  judgment  degrades.  Relevant  factors  may  be  over- 
looked; recent  information  is  often  afforded  inordinately 
higher  value  than  earlier  but  still  relevant  information. 

It  is  also  the  case  that  with  multiple  factors  exceeding 
five  to  seven  (or  less)  mental  integration  of  values  can  be 
faulty.  These  and  other  judgmental  fallibilities  can  lead 
to  decisions  that  are  often  not  consistent  with  either 
objective  information  or  the  way  that  information  is  valued 
by  the  decision  makers  (References  3,  5,  12,  14).  However, 
it  should  be  noted  that  under  some  circumstances,  highly 
trained  judges  can  indeed  make  accurate  judgments  integrating 
upwards  of  ten  factors  (Reference  10). 

The  multi-attribute  utility  assessment  methodology  used 
as  the  basis  for  the  decision  aid  development  was  evolved  to 
minimize  these  judgmental  fallibilities  in  complex  decision 
tasks.  The  method  involves  two  basic  principles:  1)  dis- 
aggregate the  decision  problem  into  its  relevant  attributes 
(factors),  each  of  which  should  be  more  within  the  cognitive 
range  of  the  decision  maker  than  the  "wholistic"  judgment 
otherwise  required;  and  2)  allocate  to  a computer  the  task 
of  logically  integrating  the  elemental  judgments  of  the 
decision  maker  into  an  implied  choice. 
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3.0  DECISION  AID  DEVELOPMENT 


3.1  Criteria  and  Model  Structure 


The  first  step  in  developing  an  aid  for  the  decision 
context  under  consideration  was  to  define  an  appropriate 
problem  structure.  In  doing  this,  we  were  guided  by  the 
following  criteria: 

1.  The  elemental  judgments  required  (dictated  by  the 
problem  structure)  had  to  be  sufficiently  few  in 
number  to  require  no  more  than  one  hour  to  com- 
plete. 

2.  The  required  judgments  had  to  be  meaningful  to 
people  knowledgeable  about  division-level  tactical 
operations . 

3.  The  structure  had  to  be  devised  to  avoid  redundan- 
cies among  factors  (to  avoid  redundant  weightings 
in  the  aggregation  of  weighted  scores). 

4.  The  structure  of  the  model  was  intended  to  ensure 
general  applicability  to  a wide  range  of  tactical 
operations,  both  offensive  and  defensive. 

Unfortunately,  there  is  no  set  of  rules  to  guide  the 
process  of  decision-problem  structuring.  It  is,  quite 
frankly,  an  iterative,  cut-and-try  process. 

To  identify  the  array  of  factors  that  influence  tactical 
choice,  relevant  Army  manuals  were  reviewed,  and  consulta- 
tions were  held  with  combat-experienced,  division-level 
commanders,  as  well  as  with  members  of  the  faculty  of  the 
Army  Command  and  General  Staff  College  at  Fort  Leavenworth. 
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From  the  array  of  relevant  factors  garnered  from  these 
sources,  preliminary  factor  structures  were  generated  and 
evaluated  by  personnel  experienced  in  the  tactical  decision 
role  at  issue.  Through  a series  of  iterations,  the  decision 
model  presented  in  Figure  1 was  evolved  as  a suitable  repre- 
sentation. Figure  2 shows  a detailed  delineation  of  the 
twenty-four  factors  contained  in  the  model  in  exactly  the 
form  they  are  posed  for  judgment  elicitation  purposes. 

3. 2 Operation  of  the  Decision  Aid 

There  are  four  basic  steps  required  of  the  decision 
maker  in  using  the  prototype  decision  aid. 

1.  For  each  of  the  factors  shown  in  Figure  2 the 
decision  maker  is  required  to  assign  scores  to 
each  of  the  courses  of  action,  designating  which 
is  best  in  terms  of  the  factor  under  consider- 
ation, which  is  worst,  and,  if  there  are  more  than 
two  courses  of  action,  where  each  of  the  remaining 
courses  of  action  would  fall  on  a scale  relative 
to  the  poorest  and  best  options.  Rules  that  must 
be  adhered  to  in  scoring  courses  of  action  are 
presented  in  Figure  3. 

2.  The  second  step  in  exercising  the  decision  aid 
requires  the  decision  maker  to  judge  for  each 
factor  the  importance  of  the  difference  between 
the  course  of  action  scored  poorest  and  best  and 
to  rank-order  the  factors  in  terms  of  these 
difference  judgments.  This  is  a convenience  step 
and  is  not  a computational  necessity.  The  rank- 
ordering step  is  intended  to  make  the  assignment 
of  importance  weights,  as  required  in  the  next 
step,  more  orderly  and  reliable. 


Descriptions  of  earlier  versions  of  the  model  are  available 
from  Army  Research  Institute,  5001  Eisenhower  Avenue,  PERI-OS, 
Alexandria,  VA  22333. 
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MISSION 

ACCOMPLISHMENT 


Fields  of  Fire 

Cover  & Concealment 

Mobility 

Sieze/Deny  Key  Terrain 
Observation 

Natural/Artificial  Obstacles 

Disposition 
Strength  & Condition 
Reserves 

Logistic  Support 
Probable  Actions/Reactions 
Command  & Control 

Current  Disposition 
Strength  & Condition 
Reserves 
Logistic  Support 
Command  & Control 

Observa  tion/Visibili  ty 
Cover /Concealment 
Mobility 

Enemy  Actions/Reactions 
Dependence  on  Other 
Commands 

Dependence  on  Surprise/ 
Deception 

Unexpected  Weather 


Figure  1.  STRUCTURE  OF  DECISION-AIDING  MODEL 


ARMY  EVAL  V 


Factor  Listing 


I.  Terrain  Factors 

As  related  to  mission  accomplishment , score  each  of 
your  courses  of  action  in  terms  of  how  well  each: 

1.1  Exploits  field  of  fire  afforded  by  terrain  features 

1.2  Exploits  cover  and  concealment  afforded  by  terrain 
features 

1.3  Exploits  mobility  provisions  due  to  terrain  features 

1.4  Accomplishes  rapid  seizure  or  denial  of  key  terrain 

1.5  Exploits  observation  provisions  of  terrain 

1.6  Exploits  or  accommodates  natural  and  artificial 
obstacles 

II.  Enemy  Force  Factors 

As  related  to  mission  accomplishment,  score  each  of 
your  courses  of  action  in  terms  of  how  well  each  exploits 
what  you  know  or  estimate  about: 

2.1  Enemy  disposition 

2.2  Enemy  strength  and  condition 

2.3  Enemy  reserves 

2.4  Enemy  logistic  support 

2.5  Probably  enemy  actions/reactions 

2.6  Enemy  command  and  control  capabilities/vulner- 
abilities 


Figure  2.  ARMY  EVAL  V 
Factor  Listing 
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III.  Own  Force  Factors 

As  related  to  mission  accomplishment,  score  each  of 
your  courses  of  action  in  terms  of  how  well  each  exploits  or 
accommodates: 

3.1  Own  force  current  disposition 

3.2  Own  force  strength  and  condition 

3.3  Own  force  reserves 

3.4  Own  logistic  support 

3.5  Own  command  and  control  capabilities/vulner- 
abilities 


IV.  Weather  Factors 

As  related  to  mission  accomplishment,  score  each  of 
your  courses  of  action  in  terms  of  how  well  each  exploits: 

4.1  Observation/visibility  conditions  forecast  to 
exist  due  to  weather 

4.2  Cover  and  concealment  conditions  forecast  to 
exist  due  to  weather 

4.3  Mobility  conditions  forecast  to  exist  due  to 
weather 


V.  Risk  Factors 

As  related  to  mission  accomplishment,  score  each  of 
your  courses  of  action  in  terms  of: 

5.1  Ability  to  cope  with  surprises  in  terms  of  enemy 
strength  or  enemy  actions/reactions 

5.2  Freedom  from  dependence  on  forces  not  under  own 
control 

5.3  Freedom  from  critical  dependence  on  surprise  or 
deception 

5.4  Suitability  under  unexpected  adverse  weather 
conditions 


Figure  2 (Con't) 
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Course  of  Action 


of  the  difference  between  the  extreme  courses  of  action  on  a 
factor.  With  practice,  the  correct  frame  of  reference  for 
judging  importance  weights  becomes  routine.  Even  then, 
however,  there  remains  an  uncertainty  band  around  the  impor- 
tance weights  entered  into  the  model.  The  decision  maker 
may  have  entered  a weight  of,  say,  70  for  a factor  but  was 
really  uncertain  as  to  whether  that  value  might  just  as  well 
have  been  60  or  80.  When  such  uncertainty  enters  the  picture 
(as  is  usually  the  case) , it  is  of  great  importance  to  know 
whether  variation  of  the  judgmental  inputs  within  the  decision 
maker's  band  of  error  would  shift  the  indicated  course  of 
action  selection  from  one  option  to  another--a  matter  of 
sensitivity  testing. 

3. 3 Sensitivity  Analysis 

A common  approach  to  a sensitivity  analysis,  which 
involves  testing  how  robust  the  ordering  of  courses  of 
action  would  be  when  weights  are  varied  within  their  toler- 
ances, is  to  vary  each  weight  individually,  keeping  the 
ratio  of  all  the  other  weights  fixed.  The  disadvantage  of 
this  approach  is  that  some  variation  of  several  weights 
together  may  cause  a change  in  the  best  course  of  action 
much  sooner  than  if  weights  were  varied  separately. 

To  overcome  this  problem,  a novel  approach  to  sensi- 
tivity analysis  in  this  kind  of  model  has  been  incorporated 
in  the  prototype  decision  aid.  Given  any  particular  set  of 
weights,  a minimum  change  in  those  weights  (moving  in  combi- 
nation) is  determined  which  would  yield  a different  course 
of  action  with  the  highest  weighted  score.  Some  measure  of 
what  is  meant  by  the  size  of  a change  in  weights  is  necessary 
to  carry  this  out;  the  measure  adopted  is  the  Euclidean  dis- 
tance, the  sum  of  the  squares  of  the  difference  of  each 
weight  from  its  starting  position.  It  is  recognized  that 
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this  nay  not  be  the  most  appropriate  measure  in  all  circum- 
stances. For  example,  if  the  user  is  far  more  confident 
with  some  weights  than  with  others,  it  may  make  sense  to 
consider  a small  change  in  a well-defined  weight  to  be 
"equivalent"  to  a much  larger  change  in  another  weight, 
suggesting  that  a weighted  sum  of  squares  may  be  a more 
appropriate  measure.  In  this  effort,  however,  issues  of 
this  kind  are  not  pursued. 

Once  the  minimum  change  in  weights  necessary  to  change 
the  recommended  course  of  action  is  determined  (along  with 
the  changed  weights) , the  decision  maker  can  consider  whether 
or  not  these  changed  weights  are  consistent  with  the  intended 
judgments.  If,  as  will  generally  be  the  case,  the  changed 
weights  are  not  consistent,  then  the  user  can  be  confident 
that  the  numbers  given,  though  rough,  have  a clear-cut 
implication.  If,  on  the  other  hand,  the  new  weights  are 
consistent  with  the  judgments,  then  it  can  be  assumed  the 
user  is  not  discriminating  between  the  two  courses  of  action. 
At  this  point,  judgments  must  be  refined  by  further  intro- 
spection and  information  gathering,  or  it  must  be  recognized 
that  there  is  no  basis  to  prefer  one  course  of  action  over 
its  closest  rival. 

As  implemented  on  the  IBM  5100  computer,  the  sensiti- 
vity test  is  performed  in  a matter  of  seconds  and  is  an 
indispensible  part  of  the  evaluation  model.  The  formal 
algorithm  underlying  the  sensitivity  test  is  presented  in 
Appendix  A. 

3 . 4 Computer  Implementation 

The  model  as  described  in  the  preceding  sections  was 
implemented  in  prototype  form  on  an  IBM  5100  portable  com- 
puter. The  computer-based  model  is  designed  for  direct,  on- 


16 


3. 


After  considering  again  the  importance  of  the 
difference  between  the  courses  of  action  scored 
poorest  and  best  on  each  factor,  the  decision 
maker  is  next  required  to  assign  relative  impor- 
tance weights  to  each  of  the  differences.  Using 
as  a reference  a weight  of  100  assigned  to  the 
most  important  difference,  the  decision  maker  must 
assign  a relative  weight  to  the  second  most  impor- 
tant difference,  the  third  most  important,  and  so 
on,  each  weight  reflecting  the  percentage  worth  of 
the  difference  under  consideration  to  that  judged 
most  important  and  assigned  a weight  of  100.  In 
making  this  judgment,  the  decision  maker  asks, 
"Relative  to  the  top-ranked  difference  and  its 
weight  of  100,  how  much  is  the  difference  between 
the  poorest  and  best  course  of  action  on  the  next- 
ranked  factor  worth  to  me,  80%  as  much?  50%  as 
much?"--and  so  on. 

The  foregoing  steps  are  accomplished  in  serial 
sets,  first  for  the  category  concerned  with  terrain 
factors,  then  for  enemy  force  factors,  own  force 
factors,  weather  factors,  and  risk  factors. 

4.  After  all  the  above  scores  and  weights  have  been 

entered  by  the  decision  maker,  the  final  judgmental 
operation  required  is  to  assign  importance  weights 
to  each  of  the  five  categories.  To  accomplish 
this,  a set  of  five  factors  consisting  of  the  top- 
weighted  factor  from  each  category  is  presented. 

Again,  the  decision  maker  is  required  to  judge  the 
relative  importance  of  the  magnitude  of  the  difference 
between  the  courses  of  action  scored  poorest  and 
best  on  each  of  these  factors  and  to  rank  and 
weight  these  differences  just  as  before.  This 
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operation  has  the  effect  of  adjusting  factor 
weights  by  the  importance  weighting  of  the  category 
of  which  each  is  a part.  If,  for  example,  the 
three  factors  under  weather  had  been  assigned 
weights  of  IOC,  80,  and  30  on  the  basis  of  their 
relative  importance  within  the  weather  category, 
and  the  weight  then  assigned  the  weather  category 
was  50,  the  adjusted  weights  for  the  weather 
factors  would  become  50,  40,  and  15. 

The  above  scoring,  ranking,  and  weighting  steps  encom- 
pass all  of  the  judgmental  inputs  required  of  the  decision 
maker.  When  these  steps  are  completed,  the  computer  into 
which  the  judgmental  values  were  entered  automatically 
calculates  weighted  scores  for  each  of  the  courses  of  action 
under  assessment.  This  calculation  is  simple  arithmetic 
involving  summing  over  all  factors  for  each  course  of  action 
the  product  of  thu  normalized  factor  weight  times  the  normal- 
ized category  weight,  times  the  factor  score  assigned  each 
course  of  action.  Assuming  that  the  scores  and  weights  are 
valid  and  that  the  model  captures  the  salient  factors  rele- 
vant to  the  decision,  the  course  of  action  yielding  the 
highest  weighted  score  should  be  the  preferred  option. 

It  is  worth  noting  at  this  point  that  while  the  process 
of  assigning  scores  to  courses  of  action  (i.e.,  specifying 
the  one  which  is  best  on  a factor,  worst,  and  intermediate) 
is  found  to  be  an  easy  task  by  most  decision  makers,  the 
process  of  assigning  importance  weights  to  magnitudes  of 
difference  is  initially  an  unfamiliar  way  of  thinking  for 
most.  Unless  users  of  the  decision  aid  are  carefully  briefed 
and  prompted  in  early  trials,  there  is  a tendency  for  many 
to  slip  into  the  conventional  (but,  in  this  context,  erron- 
eous) pattern  of  assigning  weights  to  the  perceived  impor- 
tance of  a factor  generally,  rather  than  to  the  importance 
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line  use  by  a decision  maker.  The  necessary  instructions  in 
the  dynamics  of  the  model  and  the  judgments  required  are 
programmed  for  computer  presentation  (Appendix  B presents  a 
complete  printed  version  of  the  instructions  and  factor 
structure) . The  program  systematically  presents  the  factors 
(as  shown  in  Figure  2 and  Appendix  B)  and  elicits  the  re- 
quired judgmental  inputs  (which  the  user  enters  into  the 
machine  by  means  of  simple  keyboard  operations) . When  the 
judgmental  sequence  is  completed  over  all  twenty-four  fac- 
tors, the  computer  automatically  calculates  and  displays 
weighted  scores  for  each  course  of  action  based  on  the 
judgments  provided  by  the  user.  A sample  printout  showing 
the  summary  results  of  one  application  of  the  model  is 
presented  in  Figure  4. 

The  computer  model,  in  its  present  form,  also  has 
provision  for  a number  of  analytic  and  convenience  options 
that  should  prove  useful  to  a decision  maker.  By  a simple 
operation  of  positioning  a cursor  under  the  desired  option 
presented  in  an  option  menu  (see  Figure  5)  , the  user  can 
exercise  any  of  the  following  analytical  features: 

1.  Display  Results:  Selection  of  this  option  pre- 
sents a visual  display  of  summary  results  which 
includes  cumulative  weighted  scores  for  each 
course  of  action  within  each  of  the  five  cate- 
gories, total  weighted  score  for  each  course  of 
action,  relative  weights  assigned  each  category, 
and  the  percentage  of  the  total  weight  assigned 
each  category. 

2.  Sensitivity  Test:  Selection  of  this  option 
causes  execution  of  the  sensitivity  test  routine 
as  described  in  Section  3.3  and  Appendix  A.  A 
sample  printout  showing  the  substance  and  format 
of  a sensitivity  test  is  shown  in  Figure  6. 
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1.  MISSION  ACCOMPLISHMENT 


FACTOR 

wr 

C A 1 

CA2 

CA3 

CUMWT 

1 ) 

TERRAIN 

( 9 5) 

a 

18 

1j 

23.19% 

2 ) 

ENEMY  FORCES 

< 70) 

0 

1? 

17 

17 . 09% 

3) 

OWN  FORCES 

( 1 0 0 > 

31 

43 

14 

50 . 45% 

4 ) 

WEATHER 

( 30) 

5 

7 

4 

9 . 28% 

rj> 

RISK  FACTOR 

f ' 

C.) 

( 0) 

0 

0 

0 

.0  0% 

TOTAL 

44 

79 

51 

1 0 0 .0  0% 

a.  Numbers  in  parentheses  indicate  the  weight 
assigned  each  category. 

b.  Numbers  under  CA1,  CA2,  CA3  (courses  of  action) 
are  cumulative  weighted  scores  over  all  factors 
within  each  category. 

c.  The  column  headed  "CUMWT"  shows  the  normalized 
weights  assigned  each  category. 


Figure  4.  Sample  Printout  Showing  Format 
and  Content  of  Decision  Model  Results  Calculation 


i 


SELECT  ANY  SINGLE  OPTION  BY  TYPING  A CHARACTER  UNDER  ITS  POINTER 


I'  ""DISPLAY  RESULTS 
I f”  SENSITIVITY 


| EDIT  VALUES 
I f "PRINT  RESULTS 
i I (“"RANK  RESULTS 
I I I ( " "LOAD  MODEL 


I I r SAVE  MODEL 
I I I T ” "NEW  VALUES 
I ..  _ I _ _ I _ _ I _ 


b 


Figure  5.  Sample  Printout  of 
the  Men-,  cf  Analytic  Options  Available 
in  the  Computer-Based  Decision  Aid 
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1.  MISSION  ACCOMPLISHMENT 
1)  TERRAIN 

:>)  enemy  forces 

3)  OWN  FORCES 

4)  WEATHER 

5)  RISK  FACTORS 


FACTOR 

( 1) 

( 2) 

< 3) 

( 4) 

CUR  CLJMUTS 

23. 19 

17.09 

50 .45 

9.28 

EQL  CUMWTS 

30  16 

39 . 22 

3 , 14 

15.80 

( 5)  TOTAL  HE 
.00  100.00 
11.69  100.00 


a.  The  numbers  in  the  row  opposite  "FACTOR"  represent 
the  category  titles  listed  immediately  above  that 
row;  (1)  terrain,  (2)  enemy  forces,  etc. 

b.  "CUR  CUMWTS"  shows  the  normalized  weight  assigned 
each  category. 

c.  "EQL  CUMWTS"  shows  the  normalized  weight  that 
would  have  to  be  assigned  each  category  in  combi- 
nation to  shift  the  indication  of  preferred  choice 
(best  CA)  to  the  next  closest  option. 


Figure  6.  Sample  Printout  Showing  Format 
and  Content  of  the  Sensitivity  Test  Calculation 


o o CO 


3.  Edit  Values:  Selection  of  this  option  allows  a 
user  to  return  to  any  designated  point  in  the 
model  to  change  values  and  to  recompute  results 
based  on  the  new  values.  Either  scores  or  weights 
may  be  changed  as  specified  by  the  user. 

4.  Print  Results:  Selection  of  this  option  causes 
creation  of  a comprehensive  printout  of  the  com- 
plete model,  including  scores,  weights,  and 
summary  results. 

5.  Rank  Results:  Selection  of  this  option  results  in 
the  generation  of  complete  factor  listing  in 
descending  rank-order  by  the  weight  assigned  each 
factor. 

6.  Load  Model:  Selection  of  this  option  causes  the 
display  of  titles  to  all  models  stored  on  the 
program  cassette  and  further  enables  the  user  to 
designate  any  one  of  those  models  for  loading  into 
the  computer  for  further  reference  and  analysis. 

7.  Save  Model:  Selection  of  this  option  enables  a 
user,  upon  completion  of  a model,  to  store  the 
model  on  the  program  cassette  for  further  reference. 
Models  thus  saved  are  recoverable  via  the  "Load 
Model"  option  described  in  6,  above. 

8.  New  Values:  Selection  of  this  option  calls  into 
play  the  basic,  unvalued  elicitation  form  of  the 
model. 
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4.0  EVALUATION  OF  THE  PROTOTYPE  AID 


An  informal  evaluation  of  the  decision  aid  was  conducted 
to  screen  out  any  gross  errors  in  structure,  wording,  or 
instructions  and  to  obtain  general  user  reactions  from  Army 
officers.  The  goal  of  this  evaluation  was  to  identify  any 
obviously  necessary  changes  in  preparation  for  a more  formal 
preliminary  evaluation  at  Fort  Leavenworth. 

Trial  runs  of  the  decision-aiding  model  were  conducted 
by  using  five  Army  Officers  (grades  0-4  and  0-5)  as  partic- 
ipants. The  decision  context  involved  choosing  among 
alternative  courses  of  action  in  two  separate  scenario-based 
exercises  used  for  training  purposes  by  the  Army  Command  and 
General  Staff  College,  Fort  Leavenworth.  One  of  the  exer- 
cises was  a defensive  scenario  (Jayhawk,  lesson  plan  P-113); 
the  other  involved  an  offensive  scenario  (Bonerland,  lesson 
plan  P-111) . For  purposes  of  gaining  some  insight  into  the 
reliability  of  the  judgmental  inputs  to  the  model,  each  of 
the  officers  conducted  the  same  decision  exercise  twice  in 
immediate  succession. 

Observations  and  comments  are  summarized  below: 

1.  Users  required  about  one  hour  to  complete  the 
procedure  on  their  first  exposure,  but  only  about 
35  minutes  on  their  second.  The  need  for  less 
time  can  be  attributed  to  an  increased  familiarity 
with  the  scenario  materials  as  well  as  the  aid 
procedures  and  operations.  A practiced  user  would 
probably  require  no  more  than  15  minutes. 

2.  Assigning  scores  to  the  courses  of  action  caused 
users  little  difficulty.  However,  users  did  have 
difficulty  understanding  and  applying  the  weighting 
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rules.  The  main  problem  seemed  to  be  that  the 
participants  were  unable  to  consider  the  impor- 
tance of  a factor  as  a discriminator  among  courses 
of  action.  However,  in  spite  of  this  difficulty, 
the  variability  in  weights  assigned  was  never 
great  enough  to  cause  a change  in  the  rank  order 
of  the  courses  of  action. 

3.  No  participants  recommended  deletion  or  addition 
of  factors. 

4.  All  participants  agreed  that  the  aiding  procedure 
helps  them  to  understand  their  own  judgment  struc- 
ture and  would  be  a valuable  training  device. 

While  one  felt  the  aid  was  too  tedious,  the  other 
four  felt  that  it  would  be  useful  as  an  adjunct 
field  decision  aid. 

A more  elaborate  evaluation  involving  eleven  officers 
was  conducted  at  Fort  Leavenworth.  In  general,  the  results 
substantiated  those  listed  above.  While  there  were  no 
recommendations  for  the  modification,  addition,  or  deletion 
of  factors,  two  problem  areas  in  the  use  of  the  aid  were 
identified.  First,  while  users  had  little  difficulty  under- 
standing the  concept  of  assigning  value  scores  to  the  courses 
of  action,  some  users  were  not  consistent  when  making  repeti- 
tive judgments.  Second,  there  was  considerable  difficulty 
in  applying  the  weighting  rules  requiring  the  assessment  of 
the  factor  as  a discriminator  among  the  courses  of  action. 
However,  these  problems  were  largely  overcome  with  increased 
practice  and  instruction.  All  participants  felt  the  aid 
would  be  beneficial  to  either  a field  operating  command 
staff  or  students  training  in  tactical  decision  making. 
Details  and  discussion  of  the  evaluation  are  available  in 
Appendix  C. 
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In  terms  of  continued  evaluation  and  implementation  of 
the  aid,  the  evaluation  indicated  two  promising  approaches. 

One  would  be  to  incorporate  the  aid  into  an  ongoing  tactical 
simulation  in  which  a commander  and  staff  could  actually  use 
the  aid  in  making  their  decisions.  A second  approach  involves 
using  the  aid  as  a training  device  in  teaching  tactical 
decision  making.  Students  could  work  through  the  aid  and 
compare  their  input  values  and  weights,  as  well  as  solutions, 
with  the  instructor's.  This  would  be  helpful  not  only  in 
identifying  errors  but  also  by  providing  a focus  for  dis- 
cussing factors  on  which  there  is  disagreement.  Based  on 
the  present  evaluation,  the  structure  and  format  of  the  aid 
seem  quite  adequate.  Directions  for  any  further  modifications 
will  appear  only  after  the  aid  has  been  implemented  and  used 
in  either  a classroom  or  tactical  decision-making  context. 

However,  even  without  any  modifications,  the  aid  should 
provide  a future  user  with: 

1.  A checklist  of  the  salient  factors  that  need  to  be 
considered  in  choosing  among  alternative  courses 
of  action,  thus  at  least  inviting  detailed  con- 
sideration of  the  complete  spectrum  of  factors 
that  bear  on  tactical  choice. 

2.  A logical  method  for  integrating  subjective  ele- 
mental assessments  into  an  indication  of  choice. 

To  the  extent  that  the  elemental  (factor)  judg- 
ments are  valid,  the  preferred  course  of  action 
thus  derived  should  be  the  optimal  choice  given 
the  way  the  decision  maker  has  valued  courses  of 
action  on  the  various  factors. 

3.  A means  of  well-focused  communication  about  the 
decision  problem  at  hand,  by  making  the  decision 
problem  and  value  structure  explicit  and  requiring 
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specification  of  judgments  in  quantitative  rather 
than  qualitative  terms.  As  a concise  summary  of 
how  alternative  courses  of  action  are  valued, 
there  is  opportunity  for  rapid  staff  review  and 
specific  corrective  or  amplifying  input  as  may  be 
required.  Specific  areas  of  disagreement  or  mis- 
understanding are  readily  identifiable,  and  one 
can  rapidly  test  for  the  impact  on  the  decision  of 
new  values  that  may  be  proposed. 

Since  all  training  is  basically  concerned  with  shaping 
the  trainee's  internal  value  structure,  the  property  of  the 
decision  aid  in  making  that  subjective  structure  visible  in 
detail  should  be  of  value  in  training  contexts  concerned 
with  tactical  decision  making.  Knowing  not  only  what  choice 
a student  would  make,  but  also  factor-by-factor  "why  and  how 
much"  may  well  reveal  some  astounding  misperceptions  that 
would  otherwise  go  undetected  and,  hence,  remain  uncorrected. 
Further,  in  a training  context,  prior  experience  with  other 
decision  aids  similar  in  method  has  supported  the  view  that 
the  structured  decision  model  makes  an  excellent  classroom 
tool  for  group  interaction  on  a decision  problem.  Working 
to  collectively  agree  on  values  to  enter  into  the  model 
promotes  lively  discussion  that  is  both  interesting  and  of 
apparently  high  instructional  value. 
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APPENDIX  A 

SENSITIVITY  ANALYSIS  ALGORITHM 


In  this  Appendix,  a mathematical  account  of  the  algorithm 
to  carry  out  the  sensitivity  test  is  described.  If  there  are 
n attributes  and  m courses  of  action,  the  evaluation  proce- 
dure calculates 


for  each  course  of  action  i.  Here  (w, , . . .,  w ) are  the 

i n 

weights  and  (b^,  . . .,  bin)  are  the  scores  for  course  of 
action  i.  The  best  course  of  action  is  the  one  with  the 
highest  weighted  score  b^. 


The  problem  to  be  solved  in  performing  the  sensitivity 
test  described  above  can  be  stated  as  follows.  Let  be 

the  weights  initially  elicited;  suppose  with  these  weights, 
course  of  action  p is  best.  Then  the  problem  is  to  determine 
w such  that 


n 


Z 

j-1 


2 


is  minimized,  subject  to  the  constraints 


n 


z 

j-1 


w . 


D 


bij 


n 

« Z 


w , 


j-1 


all  i + p, 


n 

Z w.  = 100,  and  0 ^ w.  < 100,  all  j. 
j = l 3 3 

This  will  find  the  nearest  point  in  w-space  to  &,  such  that  course 
of  action  p is  equivalent  to  some  other  course  of  action. 


Although  the  problem  as  formulated  is  a quadratic  program, 
and  can,  therefore,  be  solved  by  using  a quadratic  programming 
algorithm,  there  is  a more  efficient  algorithm  which  employs 
the  special  structure  of  the  problem,  and  this  has  been  used 

he  re . 
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It  is  convenient  to  talk  of  regions  of  the  feasible 
part  of  w-space  in  which  each  option  has  best  weighted  score. 
"Region  i"  is  the  part  of  w-space  in  which  E ^ >,  , for  all 

j + i.  The  problem  consists  of  finding  the  point  on  the 
boundary  of  region  p which  is  nearest  to  w.  Note  first  that 
if  the  perpendicular  distance  from  w to  the  hyperplar.e  on 
which  = E , for  some  other  option  q,  is  in  fact  on  the 
boundary  of  region  p,  then  that  perpendicular  distance  is 
a contender  for  the  minimum  distance  before  change  occurs. 

The  following  algorithm  emerges: 

1.  For  each  i + p,  calculate  the  perpendicular  pro- 
jection of  8 onto  the  hyperplane.?  w.  (bjj  - b . ) = 0 

w.=  100. 
i=l  1 

This  is  given  by  the  expression: 

o nC  . . AC 

wi  - wi  " n'B  -”A2  bpi  + iriB  - X2 

n n 2 n 

where  A = z b . , B » Z b*.,  C - Z w.b  . 

j=l  « j-l  ” j.l  J p3 

2.  Check  to  see  if  the  point  so  calculated  satisfies 
wi  i 0,  for  all  i;  if  not,  construct  a new 
problem  by  omitting  all  components  of  each  vector 
corresponding  to  indices  for  which  w^  < 0,  and 
recalculate  a similar  expression  to  the  one  above 
(with  new  definitions  of  n,  A,  B,  C,  and  subject 
this  time  to  the  constraint 

n n 

Z w.  = z w. ) . 
i=l  i=l 

Iterating  this  procedure  (several  times  if  necessary) 
will  produce  a reduced  vector  all  of  whose  elements 
are  non-negative.  A vector  in  w-space  is  now  created 
by  inserting  zeroes  for  all  components  which  have 
been  excluded  by  this  procedure. 
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The  vector  created  by  this  method  is  the  nearest 
point  from  8 to  the  hyperplane  E^  = Ep.  Let  us 
call  it 


3.  Now  check  to  determine  if  is  "covered"  in  the 


— 1 


sense  that  there  is  some  j + i or  p such  that 
n n 

E w,,_  b.,  > l 


^ik  > ^ wik  ^ik 

k*l  1K  3 k=l  1K  1K* 


If  this  is  the  case,  the  nearest  point  to  8 is 
not  in  region  i,  and  this  possibility  can  be 
excluded. 


4.  For  all  non-excluded  i p) , calculate  the 

distance  from  8 to  8^j.  The  smallest  of  these 
indicates  where  the  minimum  change  occurs,  and  it 
is  this  vector  which  represents  the  smallest 
change  in  weights  necessary  to  change  the  preferred 
option. 


This  algorithm  is  implemented  on  the  software  constructed 
for  the  IBM  5100  to  explore  the  sensitivity  of  judgments  made 
in  the  evaluation  procedure. 


APPENDIX  B 


ARMY  EVAL  V:  MANUAL  VERSION1 


This  procedure  is  designed  to  assist  you  in  evaluating 
the  different  courses  of  action  you  have  formulated  as  part 
of  the  process  of  developing  the  Commander's  Estimate.  In 
formulating  your  courses  of  action,  it  is  assumed  that  the 
following  important  steps  have  been  accomplished: 

A.  specified  tasks  identified; 

B.  implied  tasks  identified; 

C.  bounds  of  action  established  and  communicated; 

D.  all  data  relevant  to  mission/action  is  identified  and 
analyzed;  and 

E.  logical  assumptions  made  where  data  is  missing. 

With  the  above  steps  completed  and  your  alternative 
courses  of  action  formulated,  you  are  ready  to  compare  the 
courses  of  action  in  terms  of  the  many  factors  bearing  on 


1This  manual  version  of  the  decision  aid  was  prepared  to 
expedite  testing  of  the  aiding  concept  (multiple  users  could 
be  run  simultaneously) . The  version  presented  here  is  identi- 
cal to  the  computei. -based  version  with  the  following  exceptions: 

a.  Values  were  entered  via  the  computer  keyboard  in  the 
computerized  model  and  manually  noted  in  the  manual  version. 

b.  The  computerized  model  automatically  displayed  instructions 
and  the  factor  listing  (serially). 

c.  The  computer  version,  of  course,  had  the  computation  power 
described  in  Section  3.4.  There  was  no  on-line  computation 
capability  in  the  manual  version. 
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them  in  order  to  reach  a decision  as  to  which  is  best.  This 
procedure  is  designed  to  help  you  make  the  necessary  compari- 
sons and  reach  a logical  decision. 

The  procedure  works  by  asking  you  first  to  score  your 
courses  of  action  against  each, of  a series  of  factors  identi- 
fied as  being  relevant  to  mission  success.  After  you  have 
scored  your  courses  of  action,  you  will  then  be  asked  to 
rate  (weight)  the  relative  importance  of  the  differences 
between  courses  of  action  for  each  factor.  Based  on  the  ' 
scores  and  ratings  (weights)  you  assign,  an  overall  weighted- 
value  index  for  each  course  of  action  can  be  calculated. 

It  is  important  to  understand  that  this  procedure  is 
not  making  a decision  for  you.  All  the  judgments  used  are 
yours.  The  procedure,  in  this  case,  serves  as  a reminder  of 
the  array  of  relevant  factors  to  be  considered  and  provides 
a way  to  calculate  a logical  conclusion  based  on  ycur  own 
judgments. 


We  will  now  proceed  with  the  evaluation. 


RULES  FOR  SCORING 


The  first  step  will  be  to  assign  scores  to  each  of  your 
courses  of  action  for  each  of  a series  of  factors  bearinq  on 
mission  accomplishment.  The  following  rules  must  be  adhered 
to  in  scoring  your  courses  of  action: 

1.  For  each  factor,  decide  which  course  of  action  is 
best  with  respect  to  that  factor  and  assign  a 
score  of  100  to  that  course  of  action.  If  two  or 
more  courses  of  action  are  tied  as  best,  assign  a 
score  of  100  to  each  of  them. 

2.  For  each  factor,  decide  which  course  of  action  is 
worst  with  respect  to  that  factor  and  assign  a 
score  of  zero  to  that  course  of  action.  If  two  or 
more  courses  of  action  are  tied  as  the  worst  for 
that  factor,  assign  a score  of  zero  to  each  of 
them. 

3.  With  best  and  worst  identified  as  in  Steps  1 and 
2 above,  assign  relative  scores  to  each  of  the 
remaining  courses  of  action.  These  scores  should 
reflect  where  you  feel  each  course  of  action  would 
fall  on  the  scale  between  zero  and  100. 

4.  If  all  courses  of  action  are  equally  good  or  poor 
in  terms  of  the  factor  under  consideration,  assign 
a score  of  zero  to  each  of  them. 

With  these  rules  in  mind,  you  should  now  score  your 
courses  of  action  with  respect  to  terrain  factors. 


[Next  page] 
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1.  TERRAIN  FACTORS 


AS  RELATED  TO  MISSION  ACCOMPLISHMENT , SCORE  EACH  OF 
YOUR  COURSES  OF  ACTION  IN  TERMS  OF  HOW  WELL  EACH: 

1.1  exploits  fields  of  fire  afforded  by  terrain 
features 

1.2  exploits  cover  and  concealment  afforded  by 
terrain  features 

1.3  exploits  mobility  provisions  due  to  terrain 
features 

1.4  accomplishes  rapid  seizure  or  denial  of  key 
terrain 

1.5  exploits  observation  provisions  of  terrain 

1.6  exploits  or  accommodates  natural  and  artificial 
obstacles 


Scoring  Rules 


Best  CA  on  factor  = 100 
Worst  CA  on  factor  = 0 

Intermediate  CA's  = relative  score  values 
Ties  = assign  same  score 
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FACTOR  WEIGHTINGS 


It  is  now  necessary  to  assign  weights  to  each  of  the 
factors  against  which  you  have  scored  your  courses  of  action. 
The  weights  you  assign  to  each  factor  should  show  how  much 
the  difference  between  the  poorest  and  best  course  of  action 
for  that  factor  really  matters  in  terms  of  mission  accomplish- 
ment. Some  differences  between  courses  of  action  may  be 
large,  but  of  little  importance.  Others  may  be  small,  but 
highly  important.  In  assigning  weights,  you  are  to  judge 
how  important  the  improvement  is  between  the  poorest  and 
best  course  of  action  for  each  factor. 

Your  first  step  is  to  rank-order  the  factor  sheets  in 
terms  of  the  importance  of  the  difference  between  the  best 
and  worst  courses  of  action  shown  for  each  factor.  Scan  the 
factor  sheets  and  ask  yourself,  "How  much  does  the  differ- 
ence between  the  best  and  worst  course  of  action  shown  on 
the  sheet  for  each  factor  really  matter?"  Arrange  the 
sheets  from  top  to  bottom  in  descending  order  of  importance. 
The  factor  that  you  judge  to  be  the  most  important  differ- 
entiating factor  should  be  on  top. 

The  next  step  is  to  assign  weights  to  each  factor.  To 
do  this,  consider  the  topmost  factor — the  one  you  ranked 
number  1 in  terms  of  how  much  the  difference  between  the 
poorest  and  best  course  of  action  would  matter.  Enter  a 

weight  of  100  in  the  "(Weight  )"  space  on  that  sheet.  * 

Wj  th  that  as  a reference,  consider  the  factor  you  ranked  as 


In  the  manual  version  used  for  pilot  testing,  the  factor 
items  (1.1,  1.2,  etc.)  shown  on  the  preceeding  page  were  each 
contained  on  a separate  sheet  of  paper  with  spaces  provided 
for  entering  scores  and  weights.  Here,  for  ease  of  exposition, 
the  factors  have  been  consolidated  as  on  the  preceding  page. 
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second  most  important.  Ask  yourself,  "Relative  to  the  most 
important  factor  and  its  weight  of  100,  how  much  is  the 
difference  between  the  poorest  and  best  course  of  action  for 
the  second  factor  worth?  Does  that  difference  matter  only 
half  as  much?  Ten  percent  as  much?  Ninety  percent  as 
much? — and  so  on.  When  you  decide  on  its  percentage  worth 
relative  to  the  first  one,  assign  that  weight;  that  is,  if 
you  think  it  is  80  percent  as  important  as  the  first  differ- 
ence, enter  a weight  of  80,  and  so  on.  Use  the  same  thought 
process  to  assign  weights  to  all  the  factors.  Remember,  you 
are  weighting  how  much  the  difference  between  best  and  worst 
courses  of  action  for  each  factor  matters  in  terms  of  mission 
accomplishment — not  simply  how  important  a factor  might  be 
generally,  or  how  big  a difference  there  might  be  between 
cours  s of  action  for  a given  factor.  If,  in  your  judgment, 
some  of  the  factor  differences  are  equally  important,  assign 
the  same  weights  to  those  tied  factors. 

When  you  have  assigned  all  the  weights,  set  the  factor 
sheets  aside  (still  in  descending  order  of  importance)  and 
turn  to  the  next  page.  There  you  will  be  asked  to  repeat 
this  process  for  Enemy  Force  Factors. 


2.  ENEMY  FORCE  FACTORS 


AS  RELATED  TO  MISSION  ACCOMPLISHMENT,  SCORE  EACH  OF 
YOUR  COURSES  OF  ACTION  IN  TERMS  OF  HOW  WELL  EACH  EXPLOITS 
WH&T  YOU  KNOW  OR  ESTIMATE  ABOUT: 

2.1  enemy  disposition 

2.2  enemy  strength  and  condition 

2.3  enemy  reserves 

2.4  enemy  logistic  support 

2.5  probably  enemy  actions/reactions 

2.6  enemy  command  and  control  capabilities/ 
vulnerabilities 


Scoring  Rules 

Best  CA  on  factor  *=  100 
Worst  CA  on  factor  *=  0 

Intermediate  CA's  = relative  score  values 
Ties  = assign  same  score 
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3.  OWN  FORCE  FACTORS 

AS  RELATED  TO  MISSION  ACCOMPLISHMENT,  SCORE  EACH  OF 
YOUR  COURSES  OF  ACTION  IN  TERMS  OF  HOW  WELL  EACH  EXPLOITS  OR 
ACCOMMODATES : 

3.1  own  force  current  disposition 

j.2  own  force  strength  and  condition 

3.3  own  force  reserves 

3.4  own  logistic  support 

3.5  own  command  and  control  capabilities/ 
vulnerabilities 


Scoring  Rules 


Best  CA  on  factor  * 100 
Worst  CA  on  factor  = 0 

Intermediate  CA's  = relative  score  values 
Ties  *=  assign  same  score 


I 

j 
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WEATHER  FACTORS 


AS  RELATED  TO  MISSION  ACCOMPLISHMENT,  SCORE  EACH  OF 
YOUR  COURSES  OF  ACTION  IN  TERMS  OF  HOW  WELL  EACH  EXPLOITS 

4.1  observation/visibility  conditions  forecast  to 
exist  due  to  weather 

4.2  cover  and  concealment  conditions  forecast  to 
exist  due  to  weather 

4.3  mobility  conditions  forecast  to  exist  due  to 
weather 


Scoring  Rules 

Best  CA  on  factor  » 100 
Worst  CA  on  factor  = 0 

Intermediate  CA's  = relative  score  values 
Ties  = assign  same  score 
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5.  RISK  FACTORS 


AS  RELATED  TO  MISSION  ACCOMPLISHMENT,  SCORE  EACH  OF 
YOUR  COURSES  OF  ACTION  IN  TERMS  OF: 

5.1  ability  to  cope  with  surprises  in  terms  of  enemy 
strength  or  enemy  actions/reactions 

5.2  freedom  from  dependence  on  forces  not  under  your 
control 

5.3  freedom  from  critical  dependence  on  surprise  or 
deception 

5.4  suitability  under  unexpected  adverse  weather 
conditions 


Scoring  Rules 

Best  CA  on  factor  = 100 
Worst  CA  on  factor  **  0 

Intermediate  CA's  = relative  score  values 
Ties  = assign  same  score 
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WEIGHT  ADJUSTMENT 


You  have  now  scored  your  courses  of  action  against  all 
the  factors  and  weighted  those  factors  in  terms  of  how 
important  they  are  in  differentiating  between  courses  of 
action.  As  a final  step,  it  will  be  necessary  to  determine 
adjusted  weights  for  all  factors.  To  do  this,  pick  up  the 
top-ranked  factor  sheet  from  each  of  the  five  factor  sheet 
piles.  Make  certain  that  each  sheet  is  in  fact  the  one  you 
designated  as  most  important  in  its  category.  Now,  with 
these  five  factor  sheets*  in  hand,  rank-order  them  in  descend- 
ing order  of  importance  just  as  you  did  before.  Then  enter 

a weight  of  100  in  the  "Adjusted  Weight  " space  on  the 

topmost  factor  sheet.  Review  the  second  ranked  factor  sheet 
in  your  hand  and  ask  yourself,  "Relative  to  the  most  important 
factor  and  its  weight  of  100,  how  much  is  the  difference 
between  the  poorest  and  best  course  of  action  for  the  second 
factor  worth?  Does  that  difference  matter  only  half  as 
much?  Ten  percent  as  much?  Ninety  percent  as  much?" — and 
so  on.  When  you  decide  on  its  percentage  worth  relative  to 
the  first  one,  assign  that  weight;  that  is,  if  you  think  it 
is  80  percent  as  important  as  the  first  difference,  assign  a 
weight  of  80  and  so  on.  Use  the  same  thought  process  to 
assign  weights  to  all  the  factors  in  the  collection  of  the 
five  best  factors  you  are  now  considering.  Remember,  you 
are  weighting  how  much  the  difference  between  best  and  worst 
courses  of  action  for  each  factor  matters  in  terms  of  mission 
accomplishments — not  simply  how  important  a factor  might  be 


*In  the  manual  version  used  for  pilot  testing,  the  factor 
items  (1.1,  1.2,  etc.)  shown  on  the  preceding  page  were  each 
contained  on  a separate  sheet  of  paper  with  spaces  provided 
for  entering  scores  and  weights.  Here,  for  ease  of  exposition, 
the  factors  have  Deen  consolidated  as  on  the  preceding  page. 
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generally,  or  how  big  a difference  there  might  be  between 
courses  of  action  for  a given  factor.  If,  in  your  judgment, 
some  of  the  factor  differences  are  equally  important,  assign 
the  same  weights  to  those  tied  factors. 

When  you  have  assigned  all  the  weights,  we  will  calculate 
the  cumulative  weighted  scores  for  your  courses  of  action. 

The  one  with  the  highest  value  should  be  the  preferred  one 
for  the  mission  under  consideration. 
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APPENDIX  C 

AN  EVALUATION  OF  THE  PROTOTYPE  DECISION  AID 
BASED  ON  INPUT  FROM  POTENTIAL  USERS1 


A preliminary  evaluation  of  the  prototype  tactical  deci- 
sion aid,  described  in  detail  in  the  main  text,  was  conducted 
at  Fort  Leavenworth,  Kansas.  The  evaluation  focused  on 
three  issues:  (1)  the  subjective  reaction  of  potential 
users  of  the  aid;  (2)  the  impact  of  using  the  aid  on  decision 
making;  and  (3)  the  ability  of  users  to  systematically  and 
reliably  assign  numerical  values  to  their  judgments  of  the 
factor  values  and  weights. 

Two  potential  uses  of  the  decision  aid  are  as  a training 
device  for  students  of  tactical  decision  making  and  as  an 
operational  aid  for  the  division  command  staff.  To  obtain 
input  from  individuals  representing  both  of  these  areas,  the 
evaluation  was  conducted  at  Fort  Leavenworth,  where  the  ARI 
Field  Unit  solicited  participation  from  the  Command  and 
General  Staff  College  (CGSC),  Combined  Arms  Training  Develop- 
ment Activity  (CATRADA)  , and  Combined  Arms  Combat  Development 
Activity  (CACDA) . The  evaluation  was  conducted  15-18  May 
1978. 

Method 


Participants . Twelve  individuals  from  CGSC,  CACDA,  and 
CATRADA  each  participated  in  a three-hour  session.  At  the 
Lime  of  testing,  the  computer  data  for  one  individual  was 
accidentally  erased;  thus,  data  were  collected  from  only  eleven 
individuals.  The  following  biographical  data  was  also 
collected : 


J'This  evaluation  was  conducted  and  prepared  by  Ruth  H.  Phelps 
of  the  Army  Research  Institute. 
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Age : 

Years  of  Service: 
Rank : 

Education: 
Current  Position: 


31-41  years 

12-24  years 

LTC-3 ; MAJ-6;  CPT-2 

BS-5 ; MA/MS-4 ; PhD/EdD-2 ; CGSC-8 

CGSC-3;  CACDA-4 ; CATRADA-3 ; Reserves-1. 


Test  Materials*  and  Design.  The  research  design  and 
test  materials  were  developed  to  address  the  three  issues 
listed  above.  The  general  approach  to  the  evaluation  involved 
the  assessment  of  the  participants'  decision  strategies 
before  and  after  using  the  aid  as  well  as  the  assessment  of 
participants'  ability  to  provide  numerical  inputs  for  the 
aid.  The  participants'  decision  task  was  to  evaluate  and 
rank  order  three  courses  of  action  in  ti/3  context  of  a 
hypothetical  scenario. 

User  reaction.  All  participants  responded  to  a 
sixteen-item  questionnaire  concerning  their  opinions  of 
the  structure  and  phrases  used  to  describe  the  factors 
contained  in  the  aid,  as  well  as  the  aid's  potential 
usefulness  in  applications. 

Impact  of  the  aid.  The  effects  on  decision  making 
of  using  the  aid  were  assessed  by  comparing:  (1)  the 
accuracy  and  consistency  of  the  participants'  rank 
ordering  of  the  three  courses  of  action;  and  (2)  the 
number  and  type  of  factors  used  before  and  after  using 
the  aid.  In  order  to  assess  the  accuracy  and  consis- 
tency of  the  rank  ordering  (1,  above)  , participants 
listed  their  preferred  rank  ordering  of  the  three 
courses  of  action  both  before  and  after  using  the  aid. 


‘Copies  of  test  materials  are  available  upon  request  from 
Army  Research  Institute,  5001  Eisenhower  Avenue,  Alexandria, 
Virginia  22333,  ATTN:  PERI-OS. 
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In  order  to  obtain  reasonably  complete  factor  listings 
(2,  above),  participants  completed  two  additional 
tasks.  First,  they  simply  listed  from  memory  the 
"important  factors."  Second,  participants  sorted  24 
cards,  each  listing  a single  factor,  into  four  piles 
corresponding  to  the  degree  to  which  they  used  each 
factor  when  assessing  the  course  of  action.  Factor 
lists  generated  in  both  tasks  were  obtained  before  and 
after  using  the  decision  aid  the  first  time. 

Assessment  of  numerical  inputs.  The  validity  of 
the  feedback  and  numerical  products  of  the  aid  depends 
on  the  quality  of  the  numerical  inputs  provided  by  the 
user.  To  assess  the  ability  of  users  to  reliably  and 
systematically  assign  numerical  values  to  their  judg- 
ments of  factor  values  and  weights,  participants 
analyzed  alternative  courses  of  action  for  one  of  two 
different  hypothetical  scenarios.  The  scenarios  were 
selected  to  provide  diverse  contexts  in  which  to  assess 
variations  in  the  reliability  of  the  judgments.  Both 
scenarios  were  obtained  from  the  CGSC  and  are  currently 
used  for  classroom  tactical  instruction.  The  three 
alternative  courses  of  action  to  be  considered  are 
described  in  detail  by  each  scenario. 

The  Jayhawk  scenario  (Lesson  Plan  P-113)  describes 
a defensive  battle  in  detail  by  three  maps  and  several 
reports,  including  enemy  troop  lists,  weather  data, 
order  of  battle  data,  corps  analyses  of  area  of  opera- 
tions, moon  and  solar  data,  intelligence  reports,  etc. 

The  Bonerland  scenario  (Lesson  Plan  P-111)  is  an 
offensive  battle  described  by  similar  but  considerably 
less  detailed  data. 


43 


i 


To  conserve  the  amount  of  time  required  to  gather 
the  data,  two  versions  of  the  aid  were  used:  computer- 
ized and  manual.  The  structure  and  phrasing  were 
identical  for  both;  however,  for  the  manual  version 
responses  were  hand  written  and  there  was  no  capability 
for  immediate  feedback.  By  using  two  veisions,  two 
participants  could  work  simultaneously. 

Design . The  assessment  of  User  Reaction  and  Impact  of 
the  Aid  were  based  on  responses  of  all  eleven  participants. 
However,  for  the  Assessment  of  Numerical  Inputs,  six  partici- 
pants only  used  the  Jayhawk  and  four  only  used  the  Boner land 
scenario.  Half  of  the  participants  used  the  manual  version 
first  while  the  other  half  used  the  computer  version  first. 
Due  to  difficulties  in  scheduling,  participants  were  not 
counterbalanced  across  scenarios. 

Procedural  Sequence.  Each  session  lasted  approximately 
three  hours.  The  scenario  material  was  available  only 
during  steps  2,  5,  and  7 listed  below. 

1.  Presentation  of  introductory  material  to  both 
participants. 

2.  Individual  study  of  scenario  material. 

3.  Each  participant  describes  decision  strategy. 

4.  Presentation  of  decision  procedures  and  instruction 
to  both  participants. 

5.  One  participant  uses  computer  while  the  other  uses 
manual  version. 

6.  Each  participant  describes  decision  strategy. 
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7.  Participants  switch  aids  so  that  each  uses  both 
versions . 

8.  Participants  both  answer  User  Reaction  Questionnaire. 

In  order  to  complete  the  evaluation  within  the  time  constraints 
of  most  participants,  Steps  3 and  6 were  sometimes  omitted. 

The  precise  number  completing  each  section  is  stated  in  the 
Relevant  Results  section. 

Results 


User  Reaction.  All  eleven  participants  completed  the 
User  Reaction  Questionnaire. 

Irrelevant  factors.  Seven  participants  indicated 
all  24  factors  would  be  relevant  for  the  evaluation  of 
alternative  courses  of  action.  Only  two  recommended 
eliminating  factors  (#2.4,  Enemy  Logistic  Support; 

#4.1,  Observation/Visibility  Conditions  Due  to  Weather; 
#4.2,  Cover  and  Concealment  Conditions  Due  to  Weather; 
#5.2,  Freedom  from  Dependence  on  Independent  Forces). 

Two  other  respondents  felt  some  factors  were  redundant 
(#1.1,  Fields  of  Fire  Afforded  by  Terrain  Features  and 
#1.5,  Observation  Provision  of  Terrain;  #2.1,  Natural 
and  Artificial  Obstacles  and  #2.2,  Enemy  Disposition). 

Additional  and  changes  in  factors.  No  respondents 
suggested  additional  factors  be  included.  However, 
three  felt  that  the  phrasing  of  factors  #5.2  (Freedom 
from  Dependence  on  Independent  Forces)  and  #5.3  (Freedom 
from  Critical  Dependence  on  Deception)  was  unclear.  No 
other  factors  were  listed  as  ambiguous. 

Problems  in  assigning  numerical  values  and  weights. 
Five  of  the  eleven  respondents  mentioned  difficulties 
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which  could  be  generally  categorized  as  a discomfort 
with  the  weighting  rules;  this  should  be  alleviated 
with  increased  practice  and  instruction. 

Benefits  of  using  the  aid.  Seven  of  the  eleven 
participants  indicated  they  more  critically  evaluated 
the  alternative  courses  of  action  when  using  the  aid 
then  in  previous  comparable  decision  situations  when  no 
aid  was  available.  Similarly,  ten  indicated  they  con- 
sidered more  factors  in  their  evaluation  when  using  the 
aid.  In  addition,  use  of  the  aid  relieved  participants 
of  the  burden  of  remembering  all  factors  simultaneously 
(9/11)  . 

Applications  of  the  aid.  Although  all  participants 
indicated  that  decision  aids  are  helpful  and  would  use 
them  in  general,  they  felt  this  particular  aid  would  be 
of  most  benefit  in  school-based  training.  The  most 
frequently  mentioned  training  benefit  (9  respondents) 
was  "to  identify  the  factors  on  which  student  and 
instructor  disagree  and  to  focus  classroom  discussion 
in  resolving  the  conflicts."  Objections  to  using  the 
aid  in  field  operations  centered  on  its  complexity  and, 
indirectly,  on  the  amount  of  time  required.  In  addi- 
tion, five  indicated  the  most  valuable  aspects  of  the 
aid  were  captured  in  the  manual  version  (e.g.,  forces 
consideration  of  several  factors,  decomposes  problems). 

Impact  of  the  Aid.  Eight  participants  completed  the 
following  tasks: 

Preferred  rank  orders.  The  preferred  CGSC  rank 
ordering  of  the  three  courses  of  action  (school  solution) 
was  made  available  to  ARI  for  each  scenario.  Three  of 
the  four  participants  using  the  Jayhawk  scenario  agreed 
with  the  school  solution;  however,  three  of  the  four 
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using  the  Bonerland  scenario  disagreed  with  the  school 
solution.  Such  differences  may  well  reflect  differing 
prior  familiarity  with  the  scenario  since  six  had 
previously  worked  with  Jayhawk  in  the  classroom  but 
only  two  had  experience  with  Bonerland. 

The  respondents'  rank  orderings  of  the  three 
courses  of  action  was  similar  both  before  and  after 
using  an  aid.  Only  one  out  of  the  eight  switched  a 
rank  order;  there  was  no  difference  in  stability  related 
to  the  scenario  type. 

Number  of  factors.  Participants  listed  from 
memory  the  factors  they  had  actually  considered  in 
making  their  evaluation  of  the  three  courses  of  action. 
The  following  table  lists  the  total  number  of  factors 
indicated  and  the  number  of  factors  which  coincided 
with  the  24  factors  used  in  the  aid,  listed  separately 
for  two  scenarios. 


Before  Using  Aid 

After  Using  Aid 

Aid  Factors/Total 

Aid  Factors/Total 

Jayhawk 

8/10 

25/27 

Bonerland 

8/21 

16/22 

Total 

16/31 

41/49 

Clearly,  the  total  number  of  factors  listed  increased 
after  a single  exposure  to  the  aid.  It  appears  that  for  the 
Jayhawk  scenario,  the  post-aid  increase  is  due  to  the  addition 
of  factors  which  were  included  in  the  aid.  However,  for  the 
Bonerland  scenario,  the  increase  in  the  number  of  factors  is 
small  (from  21  to  22),  but  the  number  of  aid  factors  doubled 
(from  8 to  16) , indicating  the  number  of  non-aid  factors  was 
drastically  reduced.  Thus,  it  appears  that  the  increase  in 
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the  number  of  aid  factors  listed  is  a function  of  both  the 
addition  of  aid  factors  and  deletion  of  non-aid  factors. 

Participants  also  sorted  the  24  factors  into  four  cate 
qories  corresponding  to  the  degree  to  which  that  factor  was 
used:  Least,  Slightly,  Moderately,  and  Most  Used.  There 
were  no  differences  between  the  scenarios.  Below  is  a plot 
of  the  percentage  of  factors  included  in  each  category,  for 
the  two  scenarios  combined. 


Before  Aid 


Clearly,  there  was  no  change  in  the  number  of  factors 
in  the  Most  and  Moderate  categories.  The  effect  of  the  aid 
seems  to  be  an  increase  in  the  Slightly  Used  and  a correspon- 
ding decrease  in  the  Least  Used  categories.  Although  the 
increase  is  small,  it  is  impressive  to  see  any  change  following 
a single  exposure  to  an  aid. 

Assessment  of  Numerical  Inputs.  The  degree  of  consis- 
tency between  the  numerical  values  assigned  using  the  computer 
version  and  those  assigned  using  the  manual  version  was  used 
as  a measure  of  the  reliability  of  the  ratings  for  each 
participant.  Discrepancies  between  the  two  sets  of  ratings 
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indicate,  at  least  at  a gross  level,  the  degree  to  which 
users  can  systematically  provide  the  necessary  numerical 
inputs. 

Data  were  collected  for  both  the  manual  and  computer 
aids  for  ten  participants  (Jayhawk  N=6;  Bonerland  N=4). 

Calculated  rank  orders.  The  rank  order  calculated 
from  the  participants'  input  values  was  identical  for 
most  participants  in  both  the  manual  and  computer 
versions.  For  five  of  the  six  participants  using 
Jayhawk  and  three  of  the  four  using  Bonerland,  analysis 
of  their  input  values  produced  an  identical  rank  ordering 
of  the  three  alternative  courses  of  action  for  the  two 
aids . 


Value  scores.  The  values  assigned  to  each  course 
of  action  for  the  24  factors  were  matched  for  the 
manual  and  computer  aids  for  each  participant.  The 
degree  of  correspondence  between  the  two  was  assessed 
by  correlating  the  two  sets  of  values  and  assessing  the 
degree  of  discrepancy. 

The  average  correlation  between  computer  and 
manual  scores  is  r=.63.  The  correspondence  for  those 
using  Bonerland  tends  to  be  slightly  higher  (r=.71), 
than  for  those  using  Jayhawk  (r=.56). 

The  discrepancies  between  the  two  versions  can  be 
categorized  as:  (1)  reversal  of  extreme  ranks  (an 
order  of  1-2-3  becomes  3-1-2  or  2-3-1  or  3-2-1)  or  (2) 
reversal  of  middle  ranks  (1-2-3  becomes  1-3-2  or  2-1-3) . 
Clearly,  reversal  of  extreme  ranks  represents  a more 
severe  discrepancy  in  values  than  changes  in  middle 
ranks.  The  following  table  lists  the  proportion  of  all 
responses  in  each  category. 
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Perfect  Reversal  of 

Matching  Middle  Ranks 


Reversal  of 
Extreme  Ranks 


Jayhawk  (N=6)  60% 
Bonerland  (N=4)  74% 
Total  (N=10 ) 65% 


34% 

6% 

19% 

7% 

28% 

6% 

Weights . The  relative  weights  assigned  to  the  24 
factors  for  the  manual  aid  were  correlated  with  those 
for  the  computer  version  for  each  participant.  Again, 
the  average  correlations  for  the  Bonerland  scenario 
group  (r=.84)  were  higher  than  for  the  Jayhawk  group 
(r= . 72 ) . 

One  of  the  major  weighting  rule  errors  committed 
by  participants  working  with  the  Jayhawk  scenario,  but 
not  those  working  with  the  Bonerland  scenario,  was  the 
assignment  of  zeroes  to  a factor  for  all  three  courses 
of  action,  but  a greater  than  zero  weight  to  the  factor. 
(By  definition,  if  all  three  courses  of  action  are 
scored  as  zero,  the  factor  does  not  discriminate?  thus, 
the  weight  should  also  be  zero.)  For  the  six  Jayhawk 
participants,  an  average  of  17%  of  the  weights  were  in 
error.  However,  there  is  some  indication  these  errors 
might  be  eliminated  with  practice,  since  the  errors 
dropped  from  22%  on  the  first  aid  to  13%  on  the  second 
aid,  independent  of  whether  the  computer  or  manual 
version  of  the  aid  was  used  first. 

Summary  of  scenario  differences.  Although  partici- 
pants working  with  the  Bonerland  scenario  tended  to 
disagree  with  the  school  solution,  they  were  able  to 
use  the  aids  with  greater  consistency  and  fewer  errors 
in  their  assignment  of  value  scores  and  weights  than 
those  using  the  Jayhawk  scenario.  These  differences 
are  summarized  on  the  following  page: 
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Bonerland 

Jayhawk 

Value  Scores 

r=.  71 

r= . 56 

Value  Reversals 

26% 

40% 

Weights 

00 

il 

U 

r= . 72 

Weight  Errors 

0 

17% 

Agree  w/School 
Solution 

No 

Yes 

Discussion 


The  general  reaction  of  all  of  the  participants  was 
quite  positive.  Although  several  had  reservations  about  use 
of  the  aid  by  a tactical  commander,  most  thought  the  aid 
would  be  very  beneficial  for  classroom  training  in  tactical 
decision  making. 

During  the  development  of  the  aid,  there  was  some 
question  whether  users  would  have  difficulty  understanding 
and  applying  the  scoring  and  weighting  rules.  The  present 
analyses  indicate  that  there  were  indeed  some  problems,  but 
inconsistencies  were  not  large  and  users  would  be  able  to 
overcome  these  with  additional  instruction  and  practice. 
Interestingly,  there  were  consistent  differences  in  the 
number  of  errors  committed  by  participants  using  the  two 
scenarios.  Apparently  the  richer,  more  detailed  Jayhawk 
scenario  provided  a sufficiently  complex  context  such  that 
users  were  unsure,  hence  inconsistent,  about  appropriate 
value  and  weight  assignments.  This  is  a particularly  impor- 
tant finding  since  real  battlefield  situations  will  be 
exceedingly  complex  compared  to  the  scenarios;  it  may  well 
be  that  consistency  will  be  even  further  reduced  in  such 
circumstances.  However,  these  inconsistencies  may  be  out- 
weighed by  the  benefits  gained  from  a systematic  analysis 
and  evaluation  of  alternative  courses  of  action. 
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Although  there  was  little  change  in  the  orders  of  the 
preferred  courses  of  action,  there  was  an  increase  in  the 
number  of  factors  considered  after  using  the  aid.  This  is 
particularly  encouraging  since  there  has  been  previous 
concern  that  decision  makers  do  not  incorporate  sufficient 
factors  in  their  analyses.  It  is  anticipated  that  with 
extensive  practice,  users  will  become  moie  comfortable  in 
relying  on  the  aid  as  a memory  bank,  thus  allowing  them  to 
evaluate  many  more  factors. 

One  of  the  greatest  potential  advantages  of  this  decision 
aid  is  the  feedback  provided  by  the  sensitivity  analysis. 

This  feedback  indicates,  based  on  the  user's  inputs,  which 
factors  are  most  sensitive  to  changes  in  the  weight  assign- 
ments. Although  the  effect  of  providing  this  type  of  feed- 
back on  subsequent  decision  making  was  not  specifically 
evaluated,  all  participants  reported  it  would  be  helpful  in 
their  analyses  by  giving  them  better  insight  into  their  own 
judgment  structure. 

In  terms  of  continued  evaluation  and  implementation  of 
the  aid,  the  present  evaluation  indicated  two  promising 
approaches.  One  would  be  to  incorporate  the  aid  into  an  on- 
going tactical  simulation  in  which  a commander  and  staff  could 
actually  use  the  aid  in  making  decisions.  A second  approach 
involves  using  the  aid  as  a training  device  in  teaching  tacti- 
cal decision  making.  Students  could  work  through  a problem 
using  the  aid  and  compare  their  input  values  and  weights,  as 
well  as  solutions,  with  the  instructor's.  This  would  be  help- 
ful not  only  in  identifying  errors  but  also  by  providing  a 
focus  for  discussing  factors  on  which  there  is  disagreement. 

Based  on  the  present  evaluation,  the  structure  and  for- 
mat of  the  aid  seem  quite  adequate.  Directions  for  any 
further  moaif ications  will  appear  only  after  the  aid  has  been 
implemented  and  used  in  either  a classroom  or  a tactical 
decision-making  context. 
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